Digital Image Processor for a Digital Camera

ABSTRACT

A digital image process for a digital camera contains a preprocessor and a postprocessor. The preprocessor can include, for example, a uniformity corrector, a sampling filter, a modulation transformer, and a ditherer. The postprocessor can include a color interpolator, an RGB reconstruction block, a digital compressor, and a color pattern data buffer. The digital image process is programmable to operate one or more of the processing blocks of one or both the preprocessor and the postprocessor for flexible programming that can adapt to the requirements of various digital camera designs or users.

CROSS-REFERENCE TO RELATED APPLICATION

This is a continuation of U.S. patent application Ser. No. 09/087,253,filed May 29, 1998.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention generally relates to digital cameras. Moreparticularly, improved digital camera architectures and components aredescribed.

2. Description of the Related Art

Recently, digital cameras have become very popular. The digital cameraconverts an optical image to electronic image data and digitally recordsthe image data on a storage medium. When the image is reproduced, therecorded digital data is retrieved from the storage medium and displayedon a display device or printed out as a hard copy image.

Referring initially to FIG. 1, conventional digital camera systems 100typically include a digital camera 102 having an optical lens 104 forfocusing light rays onto an imaging capture unit 106. The imagingcapture unit 106 typically relies upon an array of light sensitive photooptic cells 108, capable of converting the light rays received from theoptical lens 104 into representative analog signals. Most commonly, thephoto optic cells take the form of charge-coupled devices (CCDs),although other devices such as CMOS receptors may be used as well. As iswell known in the art, each CCD array 108 must have associated with it aspecific color filter array (CFA). In most applications, the CFA is anempirically derived pattern of individual color filters each associatedwith a specific CCD cell in the CCD array. A color converter circuit 110then uses a particular interpolation algorithm associated with thespecific CFA to generate the analog signals representing the CCDmanufacturer's predetermined concept of the proper color scheme of theimage. Manual input controls 103 (i.e., push-buttons, for example)provide manual inputs to a step controller 105 suitably disposed toprovide control signals to the imaging capture unit 106. Such controlsignals are used to control such imaging parameters as f-stop, exposure,zoom, focus, and flash attachments, if appropriate.

The imaging capture unit 106 sends the analog signals representing theimage first to an analog-to-digital (A/D) converter unit 112. The AIDconverter unit 112 converts the representative analog signals intodigital signals representative of the image. The digital signals arethen passed to a digital signal processor (DSP) 114 where they areconverted to appropriate digital formats. An image compression circuit116 as well as a memory 118 both receive the appropriately formatteddigital signals. The image compression circuit 116 operates to digitallycompress the received digital images in order to reduce the amount ofresources required to further process the digital signals. One suchformatting scheme referred to as JPEG is commonly used, although thereare a wide variety of suitable picture formats. Once the image has beendigitally compressed, it is sent by way of a memory interface 120 to amemory slot 122 capable of receiving a memory card 124 suitable forstoring the compressed digital signals. Such memory cards 124 include“floppy” disks, flash EPROM cards, R/W compact disc (CD), SmartMedia andthe like.

Unfortunately, conventional digital cameras have several importantlimitations. One such limitation is the fact that the conventionaldigital camera may only use the color filter array (CFA) and itsassociated color interpolation algorithms. Any subsequent improvementsin CCD array technology cannot easily be incorporated into theconventional digital camera system.

Lack of integration also provides for expensive and cumbersome digitalcameras. As can be readily seen, the conventional digital camera system100 has many discrete functional blocks encompassing many differentcircuits. It is therefore relatively expensive for conventional digitalcameras to provide image correction, color correction, imagecompression, as well as provide all appropriate control and timingsignals in one unit.

In addition, each CCD or CMOS imager has different clockingrequirements, resolution, etc. That requires dedicated ASICs, or otherdedicated logic, to properly drive them. This results in highermanufacturing cost and an inflexible camera architecture.

It would be advantageous and therefore desirable to have availablecomponents that can be used by a variety of digital cameramanufacturers, regardless of their specific image sensor, colorinterpolation scheme, etc.

SUMMARY OF THE INVENTION

A digital image processor for use in a digital camera and methods ofusing a digital camera are disclosed. In one aspect of the invention, adigital image processor for use in a digital camera is disclosed. Thedigital camera has an image capture unit arranged to output analogsignals that represent a captured image, a converter unit for convertinganalog representations of the captured image to digital representationsof the image, a memory for storing digital representations of the image,and a processing unit capable of directing components included in thedigital camera. The digital image processor includes an input datastream processor for preprocessing images received from the converterunit and storing the preprocessed images in the memory. The digitalimage processor also includes an output data stream processor arrangedto receive preprocessed images and postprocess the retrieved images intoa viewable form.

In another aspect of the invention, a method of forming a viewablerepresentation of an image using a digital camera is disclosed. Themethod includes the following operations. First, an image is received atthe image capture unit to form a captured image followed by outputtinganalog signals representative of the captured image from the imagecapture unit. Next, the analog signals are converted to digital imagesignals representative of the captured image. Next, the digital imagesignals are preprocessed to form a digital image, such that thepreprocessing includes correcting non-uniformities in the capturedimage. Next, the preprocessed digital image is stored in a generallyaccessible memory that is part of the digital camera. Finally, thepreprocessed digital image is retrieved from the memory andpostprocessed into a viewable form.

Other aspects and advantages of the invention will become apparent fromthe following detailed description, taken in conjunction with theaccompanying drawings, illustrating by way of example the principles ofthe invention.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention is illustrated by way of example, and not by wayof limitation, in the figures of the accompanying drawings where likereference numerals refer to analogous or similar elements to facilitateease of understanding and which:

FIG. 1 is a block diagram of a conventional digital camera system;

FIG. 2A is a block diagram of a digital camera system in accordance withone embodiment of the invention;

FIG. 2B is a block diagram of an implementation of the digital imageprocessor shown in FIG. 2A;

FIG. 3 is a block diagram of a digital camera system in accordance withanother embodiment of the invention;

FIG. 4 is a block diagram of a digital image processor formed inaccordance with an embodiment of the invention;

FIG. 5 is a functional block diagram of a color interpolator inaccordance with an embodiment of the invention;

FIG. 6 is a flowchart detailing the color interpolation of a digitalimage by the color interpolation circuit in accordance with anembodiment of the invention;

FIG. 7 is a block diagram of a universal state machine controller inaccordance with an embodiment of the invention;

FIG. 8A is a block diagram of a universal state machine controller inaccordance with another embodiment of the invention;

FIG. 8B is a representative non-symmetric clock signal in accordancewith an embodiment of the invention;

FIG. 9 is a block diagram of a programmable analog reference signalgenerator in accordance with an embodiment of the invention;

FIG. 10A is a flowchart detailing authentication stamping of a digitalimage in accordance with an embodiment of the invention;

FIG. 10B is a flowchart detailing verification of anauthentication-stamped digital image in accordance with an embodiment ofthe invention;

FIG. 11 is a flowchart detailing the image processing of raw digitalimage data by a digital image processor in a digital camera system inaccordance with an embodiment of the invention;

FIG. 12 is a block diagram of a digital camera system in accordance withanother embodiment of the invention;

FIG. 13 is a block diagram of a digital camera system in accordance withyet another embodiment of the invention;

FIG. 14 is a block diagram of a digital camera system in accordance withanother embodiment of the invention; and

FIG. 15 is a block diagram of a digital camera system in accordance withanother embodiment of the invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

The present invention relates generally to digital cameras. In variousaspects of the invention, the digital camera includes a programmableprocessor capable of processing digital images. The processing mayinclude digital image correction and/or digital image authenticationstamping, a programmable source of control and timing signals and thecapability of providing an adaptive pixel color interpolator. In anotheraspect of the invention, the processor has a system bus architecturethat provides the digital camera with enhanced flexibility.

In the following description, numerous specific details are set forth inorder to provide a thorough understanding of the present invention. Itwill be apparent, however, to one skilled in the art, that the presentinvention may be practiced without some or all of these specificdetails. In other instances, well known structures or operations havenot been described in detail in order to avoid unnecessarily obscuringthe present invention.

Referring initially to FIG. 2A, a digital camera system 200 inaccordance with one embodiment of the invention will be described. Thedigital camera system 200 includes an imaging unit 202 connected to adigital image processor 204 capable of producing digital signalsrepresentative of a captured image. The digital image processor 204 isconnected to a local memory 206 and is capable of performing digitalimage processing operations upon received digital images. Suchprocessing operations include, but are not limited to, image correction,color correction, color interpolation, as well image compression and/orimage file formatting. The local memory 206 is capable of fast storageand retrieval of selected digital image files.

In one embodiment, the digital image processor 204 is also capable ofauthentication stamping a particular digital image as well as derivingcolor weight factors useful in the operation of the imaging unit 202. Amicroprocessor 208 connected to the digital image processor 204 providesdirection for the components including the digital camera system 200. Asystem memory 210 connected to the digital image processor 204 iscapable of storing files that have passed through the image processor204.

The digital images upon which the digital image processor 204 operatesmay be received from any appropriate source. Typically, raw digitalimages (i.e., those digital images that have not yet had any imageprocessing) are provided by the imaging unit 202. Since the systemmemory 210 and the local memory 206 are each capable of storing digitalimages, they too can be a source of digital images for the digital imageprocessor 204. In this way, the digital camera system 200 is capable ofproviding the user with digital images in any desired stage ofprocessing.

The digital image processor 204 is also capable of generating a widevariety of control signals and/or timing signals. Such control signalsand/or timing signals are useful in the operation of, for example, theimaging unit 202. By providing image processing as well as acting as asource of control signals and/or timing signals, the digital imageprocessor 204 reduces the number of components required in the digitalcamera system 200 thereby, for example, commensurably reducing overallcosts of manufacture of the digital camera system 200.

The digital camera system 200 also includes optics (not shown) such as,for example a lens, capable of directing light from any object to theimaging unit 202. The optics may include such optical systems as fiberoptic transducers and fiber optic cables, refractive mirror assemblies,or any system or device capable of being optically coupled to theimaging unit 202.

FIG. 2B is a block diagram of a digital image processor 250 inaccordance with an embodiment of the invention. It should be noted thatthe digital image processor 250 is but one possible embodiment of thedigital image processor 204 used in the digital camera system 200 shownin FIG. 2A. Therefore, the digital image processor 250 shall bedescribed in the context of the digital camera system 200 and allreferences included therein.

The digital image processor 250 includes a system bus 252 capable ofcarrying various signals. Such signals include but are not limited todata signals, control signals, and digital image signals. In thedescribed embodiment, the system bus 252 receives raw digital imagesfrom the image capture unit 202. The system bus 252 also carries controlsignals generated by a programmable control signal generator 254. In oneembodiment, the control signals so generated are supplied to the imagingunit 202, as needed, to control stepper motors included therein, forexample. A programmable timing signal generator 256 provides timingsignals as needed. Such timing signals are useful in the operation of,for example, image sensors included in the imaging unit 202. Inaddition, a programmable reference signal generator 258 providesreference signals. Such reference signals include those used in theoperation of, for example, any analog components included in the digitalcamera 200.

The digital image processor 250 also includes first and second imageprocessors 260, 262 connected to the system bus 252. The imageprocessors are structured such that they may provide processing inaddition to or independently of one another. It should be noted that inalternative embodiments, the digital image processors 260, 262 mayinclude any number of digital image processors as may be desired orrequired for a particular application.

A local memory interface 264 connecting the system bus 252 to the localmemory 206 provides the user of the digital camera system 200 with thecapability of quickly storing a particular digital image or series ofimages. Such digital images include those processed by the first imageprocessor 260 and/or the second image processor 262. In oneimplementation, such digital images include those raw digital imagesreceived from the imaging unit 202. Once stored in the local memory 206,the digital images are selectively available for any purpose, such asfor example, further digital image processing by any component connectedeither directly or indirectly to the system bus 252.

In one embodiment of the invention, the digital image processor 250includes an authentication stamper 266 used to generate an authenticitystamp. Typically, the authenticity stamp is useful in providingphotographers and/or other end-users with the capability of determiningthe authenticity of any digital image so stamped. A color interpolator268 is included to provide appropriate color interpolation when fullcolor interpolation of a digital image by the digital image processor250 is desired. Alternatively, when the color interpolation performed bythe color interpolator 268 is performed by external mechanisms connectedwith the digital image processor 250 (such as the microprocessor 208 ),the color interpolator 268 may be bypassed. A microprocessor interface270 provides an interface between the microprocessor 208 and the systembus 252.

Referring next to FIG. 3, a digital camera system 300 in accordance withanother embodiment of the invention will be described. It should benoted that the digital camera system 300 is but one possible embodimentof the digital camera system 200 shown in FIG. 2A. The digital camerasystem 300 includes an image capture unit 302, an A/D converter 308 thatconverts captured images to representative digital signals, and aconfigurable digital image process controller 310 arranged to performsome processing of the digital images and to provide various clockingand control signals to the image capture unit 302 and/or A/D converter308. The digital camera system 300 further includes a local memory 311for quickly storing digital images, a microprocessor 312, a systemmemory 314, a video DRAM 316 for storing video images, a video displaycontroller 317, a display 318 having a touch screen 319, and an I/0block 320.

The image capture unit 302 includes an image conditioner 304 forconditioning the light received from the optical system before it isreceived at an image sensor 306. Such conditioning can include imagefocusing, image enlargement (referred to as zooming), image exposure,and any other suitable conditioning found useful in the formation ofdigital images. The image sensor 306 typically includes a grid, orarray, of photosites. The photosites can be any device capable ofconverting incident light (in the form of photons) into useableelectrical signals. Typically, these photosites are formed of asemiconductor material and include charge-coupled devices (CCD) orcomplementary metal oxide semiconductor (CMOS) devices. Generally, eachof the photosites is associated with what is referred to in the art as apixel (short for picture element). The resolution of the image sensor306 is then determined by how many photosites are placed upon itssurface. This resolution is specified in one of two ways: by itsdimensions in pixels or by the total number of pixels in its images. Forexample, the same digital camera may claim to be 1200×800 pixels, or960-thousand pixels.

Each of the photosites in the image sensor 306 converts light into anassociated electrical signal. Typically, the associated electricalsignal so produced is related to the intensity (i.e., number of photons)of the light and not necessarily the color (i.e., frequency) of thelight that falls upon the surface of the photosite. Therefore, in orderto simulate the primary colors of red, blue, and green, for example, anassociated color filter array (CFA) (also referred to as a color mask)must be placed in proximity to the photosite array. One such CFA isshown in Table 1 for a representative 4×4 CCD array. TABLE 1 R G R G G BG B R G R G G B G B

The color filter array shown in Table 1 (known in the art as a “Beyer”pattern) illustrates one particular approach to forming a CFA where eachphotosite (and its associated pixel) has an associated single colorfilter. After exposure, the electrical signal generated by eachphotosite is converted to a digital signal taking the form of, forexample, an 8-, 10-, or 12-bit binary number, otherwise referred to as apixel color value. To create a 24-bit image, for example, interpolationis used whereby neighboring pixel color values are used to calculate thevalues for the other two primary colors for each pixel. By combiningthese two interpolated colors with the color measured by the photositedirectly, the original color of every pixel is calculated.

During operation of the image sensor 306, each of the photositesincluded in the image sensor 306 generates an associated analog(electrical) signal representative of a portion of the image rendered bythe image controller 306. The analog signals are then fed to ananalog-to-digital (A/D) converter 308. The A/D converter 308 convertsthe received analog signals representative of the captured image intocorresponding raw digital signals. An interface bus then carriescorresponding raw digital signals to the digital image processor 310. Atsome point in the processing of a raw captured image, the image istypically color interpolated by the digital image processor. As will bedescribed in more detail below, one feature of the digital imageprocessor illustrated in FIG. 3 is that it includes a programmable colorinterpolator that may be programmed to implement a wide variety ofdifferent desired color interpolation schemes.

In the embodiment shown, the digital image processor 310 is connected tothe microprocessor 312. It should be noted that a wide variety ofmicroprocessors may be used and in some implementations, it may bedesirable to combine the functionalities of the digital image processor310 and microprocessor 312. One useful function provided by themicroprocessor 312 referred to as data packetizing provides forefficient transmission and storage of the digital image data. In oneimplementation, the microprocessor 312 arranges the digital image datainto discrete units such as data words formed of n data bits. In anotherimplementation, the microprocessor 312 may packetize the digital imagedata into data packets having associated header portions and dataportions.

The microprocessor 312 is capable of directing the retrieval of digitalimages stored in either the system memory 314 or the local memory 311.Once retrieved, the microprocessor 312 is capable of directing thesending of the digital images to any number and type of output devicesthat may be provided on the camera. In one implementation, the digitalimages may be sent by way of an I/0 bus to the I/0 block 320. In thedescribed embodiment, the I/0 block 320 includes I/0 ports such as aparallel port, a serial port, a USB port, a TV signal output port, aPCMCIA port, as well as a modem port. The microprocessor 312 is alsocapable of directing the video controller 317 to store in the video DRAM316 digital video images, which are then sent via a display bus to adisplay 318 for viewing. A touch screen 319 overlaying the display 318is typically used to input display coordinate data to the microprocessor312. Such coordinate display data is useful in coordinating user inputsresulting in improved ease of use of the digital camera 300. It shouldbe noted that the microprocessor 312 is capable of parallel execution ofthese and any other instructions suitable for the operation of thedigital image processor 310.

The digital image processor 310 also provides image sensor controlsignals to the image sensor 306 by way of an optics control bus. Suchimage sensor control signals include but are not limited to thosesuitable for synchronizing the array of photosites included in the imagesensor 306. When the array of photosites in the image sensor 306 is aCCD array, the CCD array requires clock signals that synchronize, forexample, the reading of charge data in of each of the rows in the CCDarray and its associated data register. These clock signals, otherwisereferred to as pixel clock signals, may include other pixel clocksignals used to synchronize the CCD array. In addition to providing aprogrammable source of image sensor control signals, the digital imageprocessor 310 provides a programmable source of reference signalscarried by the interface bus useful in the operation of, for example,the A/D converter 308.

Referring next to FIG. 4, one embodiment of a digital image processor310 will be described. It should be noted that the digital imageprocessor 400 is but one possible implementation of the digital imageprocessors 310 shown in FIG. 3. Typically, the integrated circuit inwhich the digital image processor 400 is formed is an applicationspecific integrated circuit (ASIC) having various functional blocks andmemory blocks included therein. However, in alternative embodiments itmay be implemented in any suitable form including software andprogrammable logic and combinations of forms.

The configurable image controller 400 includes a preprocessor 402, apostprocessor 406, a local memory interface 416, a programmableuniversal controller 428 and a system CPU interface 450, all of whichcommunicate over system bus 404. Raw image data that is received by theimage controller 400 is first passed to the preprocessing stage 402,which typically does at least some preprocessing of the image databefore the image data is stored in memory, displayed or otherwisehandled.

In the illustrated embodiment, the preprocessing stage 402 includes auniformity corrector 408, a sampling filter 410, a modulationtransformer 412, and a ditherer 414. The uniformity corrector 408performs digital image uniformity correction using, for example, PhotoResponse Non-Uniformity (PRNU) correction and white balancing. Theuniformity corrector 408 is arranged to correct nonconformities in theimage sensor and typically uses a PRNU coefficient stored in the systemmemory 314 to correct any sensor based non-uniformities in the receiveddigital image. In the described embodiment, the uniformity corrector 408is placed first since it corrects sensor errors and it is expected thatalmost any digital camera incorporating the digital image processor willwant to incorporate uniformity correction. In the (believed to beunlikely) event that no uniformity correction is desired, then thescaling coefficients can be set appropriately, or alternatively, theuniformity corrector 408 can be bypassed.

After the uniformity correction has been applied, the received imagedata may be handled in a variety of ways under the direction of themicroprocessor depending upon the desires of a particular cameramanufacturer. One operational mode that may be desired is referred toherein as a “capture mode”, which rapidly stores digital images in thelocal memory 311. One appropriate capture mode may contemplate directlyoutputting images that have been processed by the uniformity corrector408 to the local memory 311. This may be particularly useful when thecamera it attempting to take a fast sequence of pictures. Alternatively,the stored digital image may be made available for immediate viewing onthe display 318.

Another operational mode that may be desired is referred to herein as a“cineview” mode. As will be appreciated by those familiar with digitalcameras, many digital cameras do not use traditional opticalviewfinders. Rather, images from the image sensor are presented at arelatively fast rate on a small display on the camera. Typically theimages are color images that are presented at rates on the order ofapproximately 16 frames per second. The sampling (decimation) filter 410provided to permit quick image size reductions may facilitate faster andmore efficient viewing on smaller displays such as those found inviewfinders. By way of example if a captured image has dimensions of1200 by 800 pixels, but the on-camera display is only 300 by 200 pixels,the amount of pixel data that needs to efficiently be transferred to theon-camera display for viewing is the smaller amount of data. Therefore,the sampling unit may be utilized to reduce the image file to theappropriate size, which speeds the processing of the image file and thedelivery of the image data to the display. The sampling filter 410receives input data from the uniformity corrector 408. The samplingfilter 410 is programmable so that the amount of decimation may beeither set by the manufacturer or programmed by the microprocessor. Thispermits the same sampling filter to be used with a wide variety ofdigital cameras, which may have very different image sensing pixelarrays and very different on-camera displays. The sampling filter mayalso be used to permit pictures having different image resolutions takenby a camera to be displayed on a single display.

While still in the cineview mode, the output of the sampling filter 410is output to the modulation transformer 412. The modulation transformer412 is connected to a ditherer 414 and is capable of correcting imagedegradation caused by the sampling filter 410. The modulationtransformer 412 outputs the corrected digital image to the ditherer 414which is arranged to perform anti-aliasing suitable to provide forbetter viewing on, for example, an LCD display. The ditherer 414 outputsthe resulting image to the system bus 404 where it is available fordisplay or even potentially, the local memory interface 416 for storingin the local memory 311. The pipelined architecture of the preprocessorpermits the lower resolution images to be quickly processed, which isparticularly desirable in the cineview mode so that the displayed imagesbetter simulate what a user might see through a conventional viewfinder.

The postprocessor 406 includes a number of processing blocks thatimplement specific transformations and other processing of an image thata camera manufacturer may desire to provide with the camera. In theembodiment shown, the postprocessor 406 includes a color interpolator422, an RGB reconstruction block 424, a digital compressor 426 and acolor pattern data buffer 427. The postprocessor is particularly usefulin processing the digital images for printing or display.

The color interpolator 422 provides color correction to the capturedimage. Specifically, as described above with reference to Table 1, eachpixel of the raw captured images typically indicates the intensity ofthe incoming light at one specific primary color as determined by thecolor filter array chosen by the manufacturer. The interpolator 422 isthen used to estimate the values of the other two colors for each pixel.To do this, color correction factors suitable for estimating the valuesof the other colors for each pixel are determined. The actual values ofthe color correction factors chosen are typically based on a variety offactors including the color filter array used, the type of interpolationdesired, and the designers sense of optimal color balance. In one aspectof the invention, the color interpolator 422 is capable of using colorcorrection factors derived by, for example, firmware in themicroprocessor 312, associated with a CFA included in the image sensor306. This feature and the structure of one embodiment of the colorinterpolator 406 will be described in more detail below with referenceto FIG. 5. By providing the capability of deriving color correctionfactors for any CFA, the configurable image process controller 310 maybe used by any digital camera manufacturer, regardless of their specificimage sensor, color interpolation scheme, etc., in a digital camerasystem. In this way, the configurable image process controller 310 willsignificantly reduce development time and costs, as well as componentcosts since specific ASICs are no longer necessary.

When not RGB based, the color corrected digital images are first outputto an RGB reconstruction block 424 and then passed to a digitalcompressor 426. Otherwise, the color corrected digital images are sentdirectly to the digital compressor 426. Such digital compressiontechniques include those techniques based upon color space conversion,such as, for example, JPEG. Once digitally compressed, the compressedimage files are then passed to the system bus 404 where, in oneimplementation, they may be stored in the system memory 314 and/or thelocal memory 311. A color pattern data buffer 427 connected to thesystem bus 404 capable of storing appropriate color interpolation inputdata is operatively connected to the color interpolator 422. Such colorinterpolation input data may include the number of pixels in the imagesensor array, the particular CFA used with the image sensor array, aswell as any particular image filtering and other appropriate digitalimage filtering values.

The described architectures provides camera manufactures with a greatdeal of flexibility in directing the data flow within the camera, aswell as in defining the camera's functionalities and designs. Forexample, a raw digital image processed by the uniformity corrector 408may be directly stored in the local memory 311 and later retrieved forfurther processing. Alternatively, the image may be passed through thesampling filter 410 and on through the preprocessor 402 prior to eitherstorage in the local memory 311 or being passed for direct viewing on aviewfinder, for example. In this case, it may be desirable to provide adata buffer (not shown) to hold the image before the sampling filter410.

Images that are stored in either memory may be retrieved and processedas desired. In some cases, the processing may be direct, while in othercases, the processing may be staged. For example, a PRNU and whitebalance processed image stored in memory may be retrieved and processedby the remainder of the preprocessor 402, including the sampling filter410, the modulation transfer function block 412, as well as thedithering block 414. In other situations, a stored image may beretrieved and passed to the postprocessor 406, authentication stamper418 or any of the other processing blocks that has direct or indirectaccess to the system bus.

In still another operational mode, the postprocessor 406 may receivedigital image files directly from the preprocessor 402 by way of thesystem bus 404. It should be noted that the preprocessor 402 and thepostprocessor 406 may concurrently process digital image filesassociated with different captured images.

In the described embodiment, the system memory 314 includes a systemdynamic random access memory (DRAM), a system read only memory (ROM),random access memory (RAM), as well as any other appropriate volatile ornonvolatile storage media. Such storage media includes but is notlimited to memory cards such as, for example, “floppy” disks, flashEPROM cards, R/W compact disc (CD), SmartMedia™ and the like.

It should be noted that due to the efficient architecture of the digitalimage processor 300, all operations are parallel in nature in that allmay be performed substantially simultaneously.

As pointed out above, the primary purpose of the postprocessor 406 is toprepare the digital images for printing or display on an externaldevice. In many circumstances, the camera's user may not have readyaccess to high quality printers suitable for printing photographs. Thus,it may be desirable for the user to send an electronic copy of aparticular captured image to a commercial entity that prints thephotograph. The postprocessor 406 can do the postprocessing necessaryfor such third party printing. However, it may be more efficiently andperfectly done by the third party which may have more sophisticatedprocessing abilities, such as the use of a more sophisticated colorinterpolator. Thus, the described camera also supports anotheroperational mode referred to herein as an off-line processing mode. Inthis mode, digital images stored in the local memory 311 or the systemmemory 314 can be output to any appropriate I/0 port included in the I/0block 320. In this way, additional digital image processing available byexternal devices may be used to complete the digital image processing.Such digital image processing may include color correction, RGBreconstruction (if necessary), MTF, dithering, etc. In this way, theuser is able to take advantage of digital image processing capabilitiesbeyond those available using the digital camera 300. In addition, bydigitally compressing the digital image(s) before being transmittedover, for example, the Internet, valuable time and resources areconserved.

Still referring to FIG. 4, the programmable universal controller 428connected to the system bus 404 is capable of selectively generatingcontrol and reference signals. Such control signals include but are notlimited to those used to in the operation of stepper motors, forexample, included in the image conditioner 304. Other signals includeperiodic signals (such as clock signals, both symmetric andnon-symmetric) used in the operation of the image sensor 306. When theimage sensor 306 includes a CCD array, these periodic signals includeclock signals referred to as pixel clock signals useful in the operationof the CCD array. It should be noted that due to the programmable natureof the programmable universal controller 428, a wide variety of imagesensors can be accommodated by the digital image processor 310 withoutthe need to resort to expensive and time consuming fitting procedures.

The programmable universal controller 428 is also capable of generatingreference signals useful in the operation of analog components includedin the digital camera 300. Such analog components for which the analogreference signals may be used include the A/D converter 308. By way ofexample, in one embodiment, the image capture unit 302 includes anautomatic rangefinder useful in determining the distance to the objectbeing photographed. Typically, the rangefinder generates signals relatedto the measured distance that are sent to an evaluator that determineswhether or not the object is within the proper range. If not,microstepper controller signals are fed back to appropriate motors thatcontrol image conditioners, such as focus and zoom. More particularly,in one example, based upon the evaluation, a focus signal generator 430and a zoom signal generator 434 generate appropriate micro-steppercontrol signals. These micro-stepper control signals are then fed to theimage conditioner 304 by way of the optics control bus. In this way, theprogrammable universal controller 428 provides for rapid and automaticfocus and zoom control without substantially increasing the number ofcomponents within the digital camera 300. Additionally, the digitalimage processor 310 is capable of reading the digital image from theimage sensor 306 and analyzing a central portion of the array ofphotosites for proper focus.

In one embodiment of the invention, a data buffer 432 associated withthe focus signal generator 430 takes the form of a look up table (LUT)having stored micro-stepper signal values corresponding to the receiveddistance signals. A data buffer 436 associated with the zoom signalgenerator 434 can also be a LUT having stored micro-stepper signalvalues associating the received distance signals to the proper zoomvalue. In addition to generating required micro-stepper controllersignals, the programmable universal controller 428 supplies periodicsignals, such as for example, timing signals. Such timing signals may begenerated in response to signals received from, for example, an f-stopcontroller, a light meter, a shutter controller as well as an associatedflash controller included in, for example, the image conditioner 304.

By way of example, the image capture unit 302 may include a photometerresponsive to the level of ambient light. The photometer may have anambient light level threshold below which signals are sent to theprogrammable universal controller 428 indicating that the light level isinsufficient to produce an image of the desired quality. At this point,a shutter timing generator 438 and an f-stop timing generator 440, forexample, generate appropriate timing signals using a clock andassociated timing data.

In one embodiment of the invention, the shutter timing generator 438 andthe f-stop timing generator 440 are associated with a shutter timingdata buffer 442 and an f-stop timing data buffer 444, respectively. Theshutter timing data buffer 442 and the f-stop timing data buffer 444 areeach capable of storing any data appropriate to the generation of therespective timing signals. A clock circuit 452 connected to the systembus 404 is used by the shutter timing generator 438 and the f-stoptiming generator 444 to generate the required control signals. In thisway, the digital image processor 400 provides an integrated automaticapproach to the operation of the digital camera system 300.

In addition to providing timing and micro-stepper control signals, theprogrammable universal controller 428 includes a universal state machinecontroller 446 capable of providing both symmetric and non-symmetricperiodic signals. Such periodic signals may include clock signals suchas those pixel clock signals used to synchronize the operation of, forexample, an exemplary CCD array included in the image sensor 306. Suchpixel clock signals include integration clocks, reset clocks, shiftclocks, and any other periodic signals deemed appropriate by, forexample, an exemplary CCD array manufacturer.

The programmable universal controller 428 also includes a universalanalog reference signal generator 448 capable of generating any requiredanalog reference signals. These analog reference signals may be used in,for example, the operation of the A/D converter 308. Both the universalanalog reference signal generator 448 and the universal state machinecontroller 446 have associated input data registers operativelyconnected to the system bus 404.

In one embodiment of the invention, the universal analog referencesignal generator 448 and the universal state machine controller 446 eachhave a set of data registers 447 and 449, respectively, for storing dataappropriate to the operational mode of the digital camera 300. By way ofexample, in what is referred to herein as the cineview mode, a stream ofdigital images are processed at a rate sufficient to simulate motion(typically in the range of approximately 10 fps to approximately 20fps). In order to provide proper control and timing signals, datasuitable for operating the digital camera 300 in cineview mode arestored in associated ones of the data registers 447 and the dataregisters 449.

The system CPU interface 450 connects the system bus 404 to themicroprocessor 312 and provides access to all internal registers anddata buffers included in the digital image processor 400. In this way,the microprocessor 312 may set all internal registers and/or databuffers as may be required for proper operation.

FIG. 5 is a functional block diagram of a color interpolation circuit500 in accordance with an embodiment of the invention. It should benoted that the color interpolation circuit 500 is but one possibleembodiment of the color interpolator 422 used in the digital imageprocessor 400 shown in FIG. 4. In the described embodiment, the colorinterpolation circuit 500 is capable of receiving a variety of colorinterpolation input data and determining the weights that are to be usedin processing raw input pixel signals into re-sampled pixel signals thathave full color at each location. The color interpolation input data mayinclude the number of photosites (or pixels) on the active surface ofthe image sensor array, the associated color filter array (CFA), as wellas any desired filtering of the re-sampled resulting image.

The color interpolation circuit 500 includes a pixel color weightgenerator 502 capable of generating the pixel color weights that, takentogether, form pixel color weight matrices used to generate there-sampled image. The pixel color weight generator 502 receives thecolor interpolation input data, such as the number of pixels in theimage sensor array, the associated CFA of the particular image sensorarray, as well as any desired filtering. The pixel color weightgenerator 502 then uses the received color interpolation input data toform a multiplexed array of pixel color weights. The multiplexed arrayof pixel color weights are then stored in a pixel color weight matrixbuffer 504 which forms an input to a configurable convolver 506. Theconfigurable convolver 506 in turn receives the raw image data andoperates to form the re-sampled resulting image having full color atevery pixel.

FIG. 6 is a flowchart detailing the color interpolation 600 of a digitalimage by the color interpolation circuit 500 in accordance with anembodiment of the invention. It should be noted that the flowchartillustrates but one possible process implemented in the colorinterpolation circuit 500 used in the digital image processor 400 shownin FIG. 5. The color interpolation of the digital image begins with thepixel weight generator 502 receiving appropriate image sensor data 610.The image sensor data may include the number of pixels in the imagesensor array as well as the particular color filter array associatedwith the image sensor array. The input data may also include optionaldigital image filtering selected by the user. Once received, adetermination is made if new pixel color weight factors are required620. If it is determined that new pixel color weight factors arerequired, they are generated 630, in one embodiment of the invention, byfirmware included in the microprocessor 312 using the image sensor inputdata. These pixel color weight factors may be generated by any number oftechniques well known to those skilled in the art. One such techniquereferred to as bi-linear interpolation uses known pixel colors to derivethe unknown pixel colors. Another well known technique referred to asbi-cubic interpolation may be used when higher color fidelity isrequired. Once the pixel color weight factors have been derived usingany suitable technique, they are stored 640 in a pixel color weightbuffer where they remain until needed. A configurable convolver thenuses the stored pixel color weight factors to process 650 receiveddigital images and generate full color digital images.

FIG. 7 is a block diagram of a universal state machine controller 700 inaccordance with an embodiment of the invention. It should be noted thatthe universal state machine controller 700 is but one possibleembodiment of the universal state machine controller 446 used in thedigital image processor 400 shown in FIG. 4. The universal state machinecontroller 700 is capable of providing both symmetric and non-symmetricclock signals as well as pulse signals. By symmetric clock signals it ismeant those clock signals having symmetric waveforms whereasnon-symmetric clock signals have non-symmetric waveforms. Such clocksignals may include those pixel clocks used to synchronize the operationof the CCD array included in the image sensor 306.

The universal state machine controller 700 includes a fast clock 702capable of generating precise symmetric waveforms. Typically, the fastclock period is approximately 10 ns but may range as low asapproximately 1 ns, or as may be required by the particular CCD arrayfor which the clock signals are being generated. In the embodiment shownin FIG. 7, the fast clock 702 forms an input to a logic unit 704 havinginput lines 706 through 712 capable of carrying input signals derivedfrom data stored in the data registers 447. The logic unit 702 uses suchinput signals to form the desired symmetric or non-symmetric clocksignal.

To form a symmetric clock signal, one implementation of the universalstate machine controller 700 provides for the input line 706 to supplyan initial state signal indicative of a high going clock signal or a lowgoing clock signal. The input line 508 supplies a period signalindicative of the number of ticks (each tick being equivalent to asingle period of the fast clock 702) for which the desired clock signalgenerated by the universal state machine controller 700 extends. Theinput line 710 supplies a first change signal indicative of the tick atwhich the polarity of the clock signal generated by the universal statemachine controller 700 changes (the half cycle of the clock signal).

If, however, a non-symmetric clock signal is desired, the universalstate machine controller 700 provides for the input line 712 to carry asecond change signal indicative of the tick at which the clock signalchanges polarity in reference to the first change signal.

It should be noted that the logic unit 704 may take the form of anyprogrammable circuit. The programmable nature of the logic block 704provides the universal state machine controller 700 with a wide range ofoperable modes.

FIG. 8A is a block diagram of a universal state machine controller 800in accordance with another embodiment of the invention. The universalstate machine controller 800 includes a counter buffer 802 connected toa fast clock 804 capable of acting as a counter. The counter buffer 802is also operably connected to a memory unit 806 capable of storing arepresentation of a desired clock signal waveform.

In operation, a representation of the desired clock signal is stored inthe memory unit 806. One such representation shown in FIG. 8A providesthe universal state machine controller 800 with data sufficient to formthe non-symmetric clock signal shown in FIG. 8B. The universal statemachine controller 800 forms the non-symmetric clock signal by applyingthe stored representation to the counter buffer 802. The counter buffer802 in turn acts in conjunction with the fast clock 804 to form theassociated non-symmetric clock signal. It should be noted that in thisparticular example, a logical “1” stored in the memory 806 correspondsto logical CLOCK HIGH while a logical “0” corresponds to a logical CLOCKLOW.

It should also be noted that the counter buffer 802 and the memory 806can be any form of memory capable of storing data consistent with theoperation of the universal state machine controller 800. Such memoriescan include but are not limited to static random access memories,dynamic random access memories, and any other suitable volatile ornon-volatile memory device.

FIG. 9 is a block diagram of a programmable analog reference signalgenerator 900 in accordance with an embodiment of the invention. Itshould be noted that the programmable analog reference signal generator900 is but one possible embodiment of the programmable analog referencesignal generator 448 used in the digital image processor 400 shown inFIG. 4. The programmable universal controller 900 includes a universalanalog reference signal generator 902 coupled to a programmable databuffer 904 capable of generating analog reference signals. It should benoted that the programmable data buffer 904 is but one possibleembodiment of the register set 449 shown in FIG. 4.

In one embodiment of the invention, the universal analog referencesignal generator 902 takes the form of a pulse width modulation (PWM)block represented as PWM 906. The PWM 906 uses data stored in theprogrammable data buffer 904 in conjunction with a fast clock 908 togenerate any desired analog reference signal. The analog referencesignals generated can be selectively formed to meet the requirements ofthe particular operating mode of the digital camera 300. In the casewhere the digital camera 300 is operating in the capture mode, theprogrammable data buffer 904 can supply data appropriate to that mode ofoperation. In another case where the digital camera 300 is operating in,for example, the cineview mode, the programmable data buffer 904 cansupply appropriate data accordingly.

FIG. 10A is a flowchart detailing authentication stamping of a digitalimage in accordance with an embodiment of the invention. It should benoted that the flowchart illustrates but one possible processimplemented in the authentication stamper 418 used in the digital imageprocessor 400 shown in FIG. 4. Therefore, the authentication stampingshall be described in context of the digital image processor 400 and allreferences included therein. First, a digital image to be authenticationstamped along with associated useful authenticating information(sometimes referred to as private camera information) are obtained 1005.The digital image to be authentication stamped and the authenticatinginformation are then processed 1010 using for example, a one-way HASHalgorithm. The resulting image digest 1015 is encrypted 1020 using asecure key to form an digital authentication stamp 1025 which isappended to the digital image 1030.

FIG. 10B is a flowchart detailing verification of an authenticationstamped digital image in accordance with an embodiment of the invention.First, a digital image having an associated authenticity stamp isreceived 1050. Next, decrypting the associated authenticity stamp usinga public key 1055 forms a first image digest. A second image digest isalso formed by processing the digital image to be verified, using forexample a one-way HASH algorithm 1060. Next, the first image digest andthe second image digest are compared 1065. Finally, verification of thedigital image based upon the comparing of the image digests is performed1070. In one implementation of the invention, if the image digests areequal, then the digital image is verified. Alternatively, if the imagedigests are not equal then the digital image is not verified.

The authenticity stamp is useful for many purposes including, forexample, authenticating the source camera, image author, and image dateof any digital image so stamped. It should be noted that the secure keyis associated with only the digital camera system 300 into which thedigital image processor 400 is installed. Any other digital camerasystem into which the digital image processor 400 is installed will havea different secure key. In this way, any digital image produced by aparticular digital camera system may be uniquely ascribed to only thatparticular digital camera system. In other implementations, theauthentication stamper 418 may include user specific information suchas, for example, user name, user address, camera serial number,manufacturing date and/or code that may be used to further identify thesource of the digital image of interest.

FIG. 11 is a flowchart detailing the image processing I 100 of rawdigital image data by a digital image processor 400 in a digital camerasystem 300 in accordance with an embodiment of the invention. The imageprocessing of raw digital image data begins with an image sensor withinthe digital camera generating a raw analog digital data that a converterthen converts to raw digital image data. A preprocessor receives the rawdigital image data 1105. The first processing stage then corrects 1110any non-uniformities in the digital image. Such non-uniformitiescorrection may be accomplished using PRNU and white balance techniqueswhere PRNU coefficients are stored in memory. Next, if the digitalcamera is operating in capture mode 1115, the corrected digital image isstored in memory 1120. If, however, the digital camera is not operatingin capture mode 1115, then it is determined whether the correcteddigital image is to be authentication stamped 1125. If the correcteddigital image is to be authentication stamped, then the correcteddigital image is forwarded to the authentication stamper 1130. If it isdetermined that the image is not to be authentication stamped 1125, thenthe corrected digital image is fully preprocessed by the preprocessor1135. Next, it is determined if the digital image is to be postprocessed on-chip 1140. By on-chip it is meant that post processing isperformed by the digital image processor. If it is determined that thepost processing is performed off-chip, the digital images are sent to anoff chip processor 1145. If it is determined that the post processing isto be done on-chip 1140, it is then determined if new pixel color weightfactors are required 1150. If it is determined that new pixel colorweight factors are required, then appropriate image sensor data isreceived 1155 and used to derive new pixel weight factors 1160associated with the image sensor. In either case, the digital image isthen color corrected 1165, and the color corrected digital image isoutput to the system bus 1170.

FIG. 12 is a block diagram of a digital camera system 1200 in accordancewith another embodiment of the invention. The digital camera system 1200is formed of the digital camera system 300 shown in FIG. 3 wherein themicroprocessor 312 has been replaced by specialized processors. Suchspecialized processors include a video processor 1202 capable ofprocessing viewable images as well as a system processor 1204 capable ofdirecting the operations of the digital camera system 1200. By providingspecialized processors, the digital camera system 1200 is well suitedfor use in more specialized applications where particular processingneeds are important. An application requiring high speed videoprocessing unavailable with more general application processors would bewell served by the digital camera system 1200.

FIG. 13 is a block diagram of a digital camera system 1300 in accordancewith yet another embodiment of the invention. The digital camera system1300 is formed of the digital camera system 300 wherein themicroprocessor 312 and the digital image processor 310 have beencombined into a microprocessor 1302. In this arrangement, the digitalcamera system 1300 has a higher degree of integration providing forfewer components in the manufacture of the digital camera system 1300.

It should also be noted that the digital image processor is capable ofsupporting any suitable number imaging arrays included in an associatedimage sensor.

FIG. 14 is a block diagram of a digital image processor 1400 inaccordance with an embodiment of the invention. The digital imageprocessor 1400 is capable of processing raw digital images from an imagesensor that includes three imaging arrays where each imaging array isarranged to respond to, for example, a single primary color. As shown inFIG. 14, the digital image processor 1400 is a straightforwardmodification of the digital image processor 400 shown in FIG. 4 and thedigital camera 300 shown in FIG. 3. The digital image processor 1400receives the raw digital images from three image sensors andsimultaneously processes the raw digital images. Three separatepreprocessors 402-1, 402-2, and 402-3 receive separate raw digitalimages from associated image sensors and process the respective rawdigital images accordingly. The processed images are then combinedusing, for example, firmware included in the microprocessor 312 beforebeing stored in the local memory 311 and/or the system memory 314. Inthis way, a camera manufacturer is capable of producing a digital cameracapable of producing very precise color photography without resorting toexpensive and time consuming fitting procedures. It should be noted thatthe authentication stamper 418 has been omitted for sake of clarityonly, and an authentication stamper may in fact be used toauthentication stamp any image the digital image processor 1400processes.

FIG. 15 is a block diagram of a digital image processor 1500 capable ofprocessing digital images from three image sensors in accordance withanother embodiment of the invention. The digital image processor 1500 iscapable of processing the raw digital images from three image processorsusing a multiplexer 1502 connected to three data buffers 1504-1 through1504-3 capable of staging raw digital images from the three imagesensors included in the digital camera. The three raw digital imagesstaged in the data buffers 1504-1, 1504-2, and 1504-3 are received atthe multiplexer 1502 based upon a selection signal generated by aselector 1506. Once selected, the raw digital image is passed to thepreprocessor 402 for suitable processing.

The invention has numerous advantages. One advantage of the invention isthat the digital image processor can be used by any digital cameramanufacturer, regardless of their specific image sensor, colorinterpolation scheme, etc., in a digital camera. The digital imageprocessor will also significantly reduce development time and costs, aswell as component costs since specific ASICs no longer must be provided.

Another advantage of the invention is that the system bus architectureprovides for flexible operation of the digital camera. The system busarchitecture also provides the digital camera manufacturer thecapability of economically configuring the digital camera as desired. Inthis way, the camera manufacturer is able to offer a wide variety ofdigital camera configurations without the need to resort to expensiveand time consuming fitting procedures.

Yet another advantage of the invention is that it provides the digitalcamera user with on demand specialized imaging modes such as the capturemode and the cineview mode. Providing these specialized imaging modesmakes the taking of quality photos using the digital camera much easier.In addition, these and other specialized imaging modes make the use ofthe digital camera more cost and time effective since the user is lesslikely to waste time and effort on photos that will not be used.

Although the foregoing invention has been described in some detail forpurposes of clarity of understanding, it will be apparent that certainchanges and modifications may be practiced within the scope of theappended claims. It should be noted that there are many alternative waysof implementing the present invention. For example, the system bus hasbeen described as carrying data signals, control signals, and digitalimage signals. In alternative embodiments, a separate control or otherbus could be provided to carry some of these signals. It is thereforeintended that the following appended claims be interpreted as includingall such alterations, permutations, and equivalents as fall within thespirit and scope of the present invention.

1-18. (canceled)
 19. A digital image processor for use in a digitalcamera having an imaging device arranged to output digital images and amemory for storing digital images, the digital image processorcomprising: a preprocessor comprising hardware for preprocessing digitalimages received from the imaging device and storing the digital imagesin the memory; a postprocessor comprising hardware arranged to receivedigital images and to postprocess the digital images into a viewableform; wherein the digital image processor is operable in a first mode inwhich data corresponding to preprocessed images from the preprocessorare directed to the memory, thereby bypassing the postprocessor; andwherein the digital image processor is operable in a second mode inwhich data corresponding to the preprocessed images from thepreprocessor are directed to the postprocessor for postprocessing anddata corresponding to postprocessed images from the postprocessor aredirected to the memory.
 20. A digital image processor as recited inclaim 19 further comprising a system bus within the digital imageprocessor, wherein the preprocessor, the postprocessor, and an interfacefor the memory are coupled to the system bus.
 21. A digital imageprocessor as recited in claim 20, wherein the postprocessor includes acolor interpolator arranged to derive an unknown pixel color valueassociated with a first pixel based upon at least one known pixel colorvalue associated with at least one other pixel using pixel color weightfactors associated with an image sensor in the imaging device.
 22. Adigital image processor as recited in claim 21, further including acolor pattern setting buffer that is connected to the color interpolatorand capable of storing image sensor data associated with the imagesensor included in the imaging device, the image sensor data being usedto derive the associated pixel color weight factors.
 23. A digital imageprocessor as recited in claim 20, wherein the postprocessor is arrangedsuch that its operation does not interfere with the operation of thepreprocessor or taking pictures using the digital camera.
 24. A digitalcamera comprising: an imaging device arranged to output digital images;a digital image processor as recited in claim 19; and memory for storingthe digital images.